Technology Closes the Gap between Students’ Individual Skills and Background Differences

We examine the correlation of 12 background variables determined from a student survey with assessment instruments including paper-testing instruments (final exam and weekly tests) and an electronic assessment from the results of a Web-based homework tutorial. Several of the initial background variables showed correlation with hand-graded weekly or final paper tests. Level of previous math and physics courses taken correlated with positive results on both. However, none of the background variables correlated with the student’s performance on the more reliable Web-based tutorial homework score. On this basis we suggest that the use of un-timed Web-based tutorials can make an important contribution to assess student performance, without bias due to students’ background differences. Purpose Assessment is a key component of the President's plan in the No Child Left Behind Act (2002) to ensure that all students receive an adequate education. Preferably, such assessment should be embedded in a normal course of instruction rather than be administered sporadically, as are examinations in most courses. In principle, the intelligent tutoring community can use tutoring data to determine students’ levels of knowledge. Even better, a good assessment should let us know not only which students are falling behind, but also the major stumbling blocks to each student's progress (Sclafani, 2004). Any electronic tutor can be programmed to provide a detailed record of each student’s interactions while working through the tutorial, and such data can be processed to provide a highly reliable assessment (Pritchard & Morote, 2002). This information can be used by the electronic tutorial to provide Just-in-Time feedback, which is the essence of Just-in-Time Teaching pedagogy, and to help students achieve mastery (Novak et .al., 1999; Gavrin, 2003). The ultimate goal of Web-based homework tutorials is to teach scientific knowledge, a solid understanding of scientific concepts, and problem-solving skills that help students master a required level of knowledge regardless of their background. This paper concerns embedded assessment in a Socratic tutorial environment, specifically a Webbased homework tutorial that has been shown to offer a positive educational benefit (Morote & Pritchard, 2002). Similar results predicted by others’ studies (Mestre, 2001, Novak et al., 1999). We contrasted tutoring data from a Web-based homework tutor with paper testing instruments such as weekly tests and a final exam. The students’ responses in the Web-based homework tutor are “observed” by this existing sophisticated electronic tutor and scored by its algorithm. The principal result of this paper is that the average final scores from a tutorial environment of students with different backgrounds such as different genders and those with prior subject knowledge are equal. In contrast, paper-testing instruments such as a final exam or weekly tests are shown to differentiate based on students’ initial backgrounds. Data sources This study was conducted using the framework of an introductory Newtonian Mechanics (8.01) course given in the spring semesters of 2001 and 2002 at the Massachusetts Institute of Technology. There were approximately 100 students in 2001 and 75 students in 2002. The pools of students in 2001 and 2002 were very similar, scoring within one-third of a point of each other on the Mechanics Baseline Pre-Test (Hestenes & Wells, 1992). This course had been designed especially for the approximate 90% of students who had previously taken this course but failed to obtain a grade of C or better. The course utilized several instructional elements, including class participation (in recitations), hand-graded written homework, group problem-solving, weekly open tutorials for all students (those students doing poorly on the previous week’s test were required to attend the tutorials), and an electronic homework tutor (myCyberTutor, EET, 2003). Paper testing included weekly tests and a final exam which evenly reviews the entire course; all were hand-graded paper tests with some partial credit given for incorrect answers. A typical student interacted with myCyberTutor approximately 48 hours/semester, averaging 10-20 minutes on each problem. Each problem involves approximately 11 interactions between student and computer. These were logged with a time stamp, giving 11 timed reference points along the student’s route to the solution. In contrast, the 12 weekly tests consumed a total of 4.5 hours, and the final exam length for both years was 3 hours.These three assessment instruments were considered in this study: Weekly test (WT). Students were given weekly hand-graded written tests on the topics of that week. These were timed tests, but with sufficient time only about 10% did not finish early. The weekly test average for the whole term is designated as WT01 for the Spring 2001 semester and WT02 for the Spring 2002. Final exam (FINAL). This is a timed final test. We note in passing that this particular final exam had three distinct segments. A standard test, Mechanics Baseline test (Hestenes & Wells, 1992), was used for one-third of the final. The remainder was divided into basic skill and complex problems which accentuate the performance of different students. The total test requested 47 specific responses from the students (compared with approximately 28 responses typical at MIT). Additional questions should make this final exam more reliable. Final01 designates the 2001 final and Final02 the 2002 final. MyCyberTutor lost points algorithm (CT). The tutoring data was asssessed with an algorithm based on the “adhoc” generalization of a typical homework scheme in which points are awarded primarily for correct work. A small penalty was subtracted for submitting incorrect answers as well as for requesting hints to discourage students from requesting hints and subproblems without thinking first. The 2001 algorithm, CT01, depended on the number of correct responses minus solutions requested, and provided a bonus of 0.03 points for each unopened hint. In addition to correct responses, solutions requested, and hints, in the Spring 2002, the CT02 algorithm also deducted 0.03 points for incorrect responses that received useful feedback. Previous studies show that students’ backgrounds have some influence on their course performance. Therefore, myCyberTutor asks twelve questions in an initial survey (Table 1). No attempt has been made to use this data to improve students’ assessment during the term, although it would probably be valuable for this. Background variables are divided in four: level of knowledge, technological experience, demographic and others: Level of knowledge The intention was to learn about students’ initial level of knowledge. They were asked for their level of experience in specific subject areas: Algebra. Students chose one of these options regarding their knowledge of Algebra: a) No experience b) 1 year of high school Algebra c) 2 years of high school Algebra d) 3 years of high school Algebra Physics. Students chose on of the following options about their experience with Physics: a) No experience b) 1 year of high school Physics c) 2 years of high school Physics, c) 3 years of high school Physics d) 1 semester of college Physics e) 2 semesters of college Physics f) More than 2 semesters of college Physics Calculus. a) No experience b) High School Calculus c) High School AP Calculus d) College Differential Calculus e) College Integral Calculus f) Multivariate Calculus TriGeo. Students were asked for their level of Geometry/Trigonometry, they had to choose one of these options: a) No experience b) High School Geometry c) High School Trigonometry d) High School Analytic/ Solid Geometry Math. Students answered the following question: “Which math course are you taking this semester?” Their answer was scored from 1 (minimum level in class) to 5 (maximum level in class) depending of the course mentioned. Technological experience Access. We wanted to know if access to a computer will influence performance. On a scale of 1-5, rate your level of access to a computer (1=easy access, 5=difficult access). OtherE. We were interested in learning if the students already had experience with Web-based learning systems. They chose one of the following options: a) no experience b) some experience c) extensive experience Comfort. This variable is intended to learn about the reaction of the students to Web-based systems. Students answered “How comfortable would you say you are with the World Wide Web?” they chose one of these options: a) very comfortable b) comfortable c) not comfortable Demographic Demographic data were included such as: “age” and “gender” Other Level. Students were asked for their current grade level : a) college freshman b) college sophomore c) college junior d) college senior e) high school freshman or sophomore f) high school junior g) high school senior Major. Students were asked for their current or intended major. Variables Description